Skip to content

perf(gfql): engage #1658 index for seeded gfql()/Cypher chains on ALL 4 engines (incl. typed edges)#1734

Merged
lmeyerov merged 5 commits into
masterfrom
feat/gfql-typed-edge-index-coverage
Jul 19, 2026
Merged

perf(gfql): engage #1658 index for seeded gfql()/Cypher chains on ALL 4 engines (incl. typed edges)#1734
lmeyerov merged 5 commits into
masterfrom
feat/gfql-typed-edge-index-coverage

Conversation

@lmeyerov

@lmeyerov lmeyerov commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Summary

A seeded hop expressed as a gfql([...])/Cypher chain (rather than a direct g.hop(...)) previously always fell back to the O(E) scan, even with a resident #1658 adjacency index. Root cause: both chain executors attach a synthetic per-edge id column to the edge frame (pandas/cuDF eager _chain_impl: copy(deep=False)+assign; polars native lazy executor: with_row_index) → a fresh edge-frame object → the index's source_ref is df identity guard missed → index_seeded_hop returned None → scan. This is the material Search lever for real Cypher/LDBC workloads (seeded traversals expressed as chains with typed edges).

Changes

  • registry.rebind_edges(new_edges) — re-points the edge adjacency indexes' identity guard at the chain's augmented frame. Safe: the shallow copy / with_row_index preserves the indexed src/dst columns by value, and the structural fingerprint (rows + bound columns + engine) is unchanged by an added column. NODE_ID left untouched. Applied at both augmentation sites: compute/chain.py (pandas/cuDF eager) and lazy/engine/polars/chain.py (polars/polars-gpu native lazy).
  • Typed edges_hop_is_index_coverable now accepts a simple scalar-equality edge_match (-[:KNOWS]-> / e_forward({"type": X})) on the wavefront path; index_seeded_hop applies the edge predicate to the CSR-matched rows each hop, parity-exact with the scan's filter_edges_by_dict.
  • Deliberately NOT covered (stay on scan, no over-reach): predicate/membership-list edge_match, edge_query, and the direct-hop (non-wavefront) path.

Validation (dgx-spark, 500k nodes / 4M edges / 4 edge-types, warm median-of-7; every cell verified index result == scan result)

shape pandas cuDF polars polars-gpu
typed 1-hop 2.1× 1.9× 1.3× engaged (1.0×)
typed 2-hop 1.4× 2.0× 1.4× 1.3×
untyped 1-hop 2.7× 2.0× 5.4× 12.4×
membership-list (control) scan ✓ scan ✓ scan ✓ scan ✓

All shapes PARITY=True (exact). All four engines now engage the index for typed AND untyped seeded chains. Small synthetic-graph typed speedups understate the real win (the reverse-pass still scans and synthetic degree is low); the index-vs-scan gap is far larger on real LDBC/Pokec shapes. Regression: test_index.py 138 passed (109 existing + 29 new differential/engagement/rebind tests), 4-engine.

🤖 Generated with Claude Code

@lmeyerov lmeyerov changed the title perf(gfql): engage #1658 index for seeded gfql()/Cypher chains on pandas/cuDF (incl. typed edges) perf(gfql): engage #1658 index for seeded gfql()/Cypher chains on ALL 4 engines (incl. typed edges) Jul 19, 2026
@lmeyerov
lmeyerov force-pushed the feat/gfql-typed-edge-index-coverage branch from 5f18f4c to 31b07ba Compare July 19, 2026 17:25
@lmeyerov
lmeyerov changed the base branch from master to perf/gfql-pl-nan-clean-cache July 19, 2026 17:25
@lmeyerov

Copy link
Copy Markdown
Contributor Author

Stacked: rebased onto #1735 (perf/gfql-pl-nan-clean-cache) and re-based the PR accordingly — this is now the second PR of the GFQL perf stack (#1735#1734 → Cypher-Search fast path). The rebase is content-identical to the prior tip modulo #1735's Engine.py/nan-clean-cache commit underneath (diff-verified); the only conflict was CHANGELOG, resolved by keeping both entries. #1733 is closed as subsumed by #1735. Land order: #1735 first, then this.

Comment thread graphistry/compute/gfql/index/registry.py
@lmeyerov
lmeyerov force-pushed the feat/gfql-typed-edge-index-coverage branch from 13d2442 to 8df8249 Compare July 19, 2026 20:45
lmeyerov and others added 4 commits July 19, 2026 14:08
…das/cuDF (incl. typed edges)

A seeded hop expressed as a gfql()/Cypher chain (not a direct g.hop) always
scanned on pandas/cuDF even with a resident index: _chain_impl attaches a
synthetic per-edge id column (copy(deep=False)+assign) -> a fresh edge-frame
object -> the index's `source_ref is df` identity guard missed.

- registry.rebind_edges: re-point edge adjacency indexes at the augmented frame
  (safe: shallow copy preserves src/dst by value; fingerprint unchanged by an
  added column). Called in chain.py right after the edge-index augmentation.
- Typed edges: relax _hop_is_index_coverable for simple scalar-equality edge_match
  on the wavefront path; index_seeded_hop applies the edge predicate to CSR-matched
  rows each hop, parity-exact with the scan's filter_edges_by_dict.
- Predicate/membership edge_match, edge_query, and the direct-hop path stay on scan.

Measured (dgx, 500k/4M/4-types, warm median, index==scan): pandas typed 2.1x /
untyped 2.7x; cuDF typed 1.9x / untyped 2.0x. polars/polars-gpu untyped already
engage via the native lazy executor; typed polars is a tracked follow-up.

27 new 4-engine differential + engagement tests; existing 109-test index suite green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…edges

The native polars lazy chain executor (lazy/engine/polars/chain.py) attaches its
synthetic edge id via with_row_index -> a fresh edge frame -> the #1658 identity
guard missed, so typed-edge chains scanned on polars/polars-gpu (untyped already
engaged). Apply the same rebind_edges fix at the polars augmentation site.

Now ALL FOUR engines engage the index for typed AND untyped seeded chain/Cypher
hops. Measured (dgx, 500k/4M/4-types, index==scan): polars typed 1.3x / untyped
5.4x; polars-gpu typed engaged / untyped 12.4x. Broadened the engagement test to
all engines; 138 index tests pass (109 existing + 29 new), 4-engine.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…y check; cast ArrayLike operands for mask &

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…nd_edges (review)

Review feedback on #1734:
- types.py gains ScalarMatchValue / EdgeMatchValue / EdgeMatch /
  SimpleEqualityEdgeMatch mirroring exactly the runtime shapes filter_by_dict
  accepts; is_simple_equality_edge_match becomes a TypeGuard so the index path
  gets real narrowing instead of Optional[dict], and _build_edge_keep_mask /
  index_seeded_hop take EdgeMatch.
- rebind_edges no longer trusts its docstring: the strong-ref swap is gated on
  the index's O(1) structural fingerprint (row count + cols + engine) and
  indexed-column presence; any mismatch DROPS the edge index (safe miss -> scan)
  instead of re-pointing it at a frame it wasn't built over. Value-level
  preservation stays the caller's promise (checking it would be the O(E) scan
  this path avoids). + enforcement test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
@lmeyerov
lmeyerov force-pushed the feat/gfql-typed-edge-index-coverage branch from 8df8249 to 11d0bab Compare July 19, 2026 21:08
… scan path (review blockers)

Review of #1734 found the guard's 'simple scalar equality' narrower than
filter_by_dict's runtime behavior — three defects in one seam, all fixed:
- F1 (BLOCKER): frozenset (+ pd.Index/pd.Series) leaked past the (list, tuple,
  set, dict) check but are membership (isin) on the scan path; a bare == is
  silently all-False -> wrong answer on public g.hop(). The guard now delegates
  to filter_by_dict._is_membership_filter_value (single source of truth);
  EdgeMatchValue widened to match.
- F2 (BLOCKER): null-carrying match columns (pandas nullable dtypes, polars
  nulls — common after NaN->null coercion) produced NA cells whose to_numpy()
  yields an object-dtype array, exploding later at rows[edge_keep[rows]]
  (IndexError). Masks now fill_null(False)/fillna(False) before materializing —
  parity-exact (null == val drops on the scan path too).
- F3 (IMPORTANT): numeric-col-vs-str / string-col-vs-numeric edge_match returned
  a silent empty subgraph where the scan raises (GFQLSchemaError E302 on
  pandas/cuDF; polars its own ComputeError). The mask builder mirrors
  filter_by_dict's exact two checks and declines -> falls back to the scan ->
  users get the SAME error as their engine's scan.
+ 3 engine-parametrized regression tests (frozenset parity, null-column
  no-crash+parity, dtype-mismatch error parity).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
Base automatically changed from perf/gfql-pl-nan-clean-cache to master July 19, 2026 23:23
@lmeyerov
lmeyerov merged commit 7163f10 into master Jul 19, 2026
75 checks passed
@lmeyerov
lmeyerov deleted the feat/gfql-typed-edge-index-coverage branch July 19, 2026 23:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant